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I . INTRODUCTION 

During the last decade, several major systems have been 
developed to improve scheduling procedures, resource management, 
and cost control of complex programs. A multiplicity of these 
management systems has been developed by the military services 
or by major contractors working with the Department of Defense 
and National Aeronautics and Space Administration. Generally, 
these systems are derivations of the "Critical Path Method" 
(CPM) or "Performance Evaluation and Review Technique" 
(PERT) . 

As the multi-media course development contract, for the 
United States Naval Academy (USNA) , required both an extensive 
research effort and an experimental materials preparation pro- 
gram, the development of an integrated management system for the 
project was undertaken. 

It was necessary to use a system which would enable the most 
efficient determination of: 

• time estimates for completion of any and all parts 
of the project. < 

• resource requirements (manpower and other resources) 
for any and all parrs of the project. 

• the effect of schedule shifts or delays on resources, 
project costs, and delivery dates. 

• the effect of variation in sequence or inclusion of 
project tasks on resources, schedules, and project 
costs. 



• on-going performance evaluation which could be used to 
continually readjust resource, time, and cost estimates 
for the project. 
It was decided that the PERT time system could best be 
adapted to form the basis of the planning and control systems for 
the multi-media project. 

The PERT system, initially developed in 1958, has proved to 
be a useful tool in planning and evaluating projects in develop- 
ment and construction. Most of these projects have been con- 
cerned with the development and construction of weapons and 
military materiels. 

A careful review of the literature would yield no indication 
that the PERT system had ever been applied to a large-scale pro- 
ject concerned with development and research in the field of 
educational psychology. 

The application of such a system to any research and devel- 
opment project is based on the presumption that sufficient experi- 
ence with the activity exists to permit the activities involved 
to be identified. Since many similar systems have been used 
extensively for efforts in technical writing and materials pre- 
paration, considerable information was available for the establish- 
ment of a large portion of the procedural steps involved in the 
project. On the other hand, many of the research activities were 
relatively new and no adequate systematic outline of procedures 
had been developed to serve as a model for the project. In 
general, developmental projects in education and projects which 
are primarily concerned with research in any field have not been 
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repeated in sufficient quantity to establish a generalized pro- 
cedural model. 

It was believed, however, that a careful analysis of the 
work necessary for the accomplishment of each milestone would 
lead to the development of a general list of p-ocedural steps. 
It is important to note that regardless" of how well (or ill), 
defined the work tasks are, deviations from the general model 
will occur. 

Developmental activities are relatively new to the field of 
education. The problem involved in the multi-media contract was 
the production of a particular kind or type of product, from the 
initial planning stages through the development of prototypes 
which can be used for future production models. 

Because no generally agreed upon procedures exist for the 
development of a multi-media course at the college level, the 
extant procedures for preparing programed instructional materials 
and film strips were used as a guide. 

The research aspect of the project can be divided into three 
major areas: (1) evaluation, (2) experimentation, and (3) valida- 
tion. 

The general nature of the evaluation research involves the 
use of existing data and the collection of new data to answer 
specific questions an^. hypotheses. Attempts were made to secure 
valid and reliable information from the USNA files, by the care- 
ful examination of historical records and contemporary progress 
reports. Additional data was to be collected by an extensive test- 
ing program at the USNA. In developing the network of activities 
associated with the development and compilation of student data. 
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the development of the original hypotheses was among the first 
steps taken. A common fault in this type of research is the in- 
adequate consideration of the purpose for the study and how the 
data ..ill eventually be analyzed. Only after the hypotheses are 
established, can the collection, storage, and manipulation of the 
data be adequately defined. 

Experimental research is quite commo.i in the fields of 
education and psychology. The general procedure is similar to 
that used in evaluation research. Initially, hypotheses are 
established. These hypotheses lead to the specification of 
variables which are to be manipulated, and the development of an 
experimenital design. 

The unique aspect of this project is the interdependency 
between the experimental design and the development of course 
materials. In order to evaluate the effects of various media and 
presentation design elements on the student's learning style, it 
was necessary for the research design to set the parameters for 
the development of the materials. For this reason, interaction 
between the two major paths of the network" was required in all 
phases of the project. 

Validation of materials after implementation was not con- 
sidered in the first phase of the project. This aspect of the 
research effort is to be developed in Phase II. 

Although the primary reason for the development of the PERT 
time system was to ensure the effective management of the multi- 
media contract, a secondary outcome of the system was important 
to Westinghouse Learning Corporation (WLC). If the PERT time 



system could be shown to effectively control the raulti -media 
course development project, it would be extended to other major 
projects of similar nature within WLC. For this reason, the cost 
of developing and implementing the system was considered a 
developmental project within WLC, and was not charged against the 
USNA contract. 

A description of the developed system and the problems 
encountered in its operation are believed to be of sufficient 
interest to be presented in detail. It is hoped that such a 
report will enable other contractors to avoid the major pitfalls 
encountered by WLC. 
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II. DEVELOPMENT OF THE SYSTEM 

OVERVIEW 

The accuracy of any management system is a function of 
the precision and clarity of the original work specifica- 
tions. The total project must first be defined and then 
subdivided into a series of major units of work. Each of 
these units represent milestones of accomplishment toward- 
the completion of the project. Milestones represent key 
network events that are necessary to achieve the total pro- 
ject objectives. 

The important steps in the PERT system, as it was de- 
veloped for the multi-media course development project, 
are discussed in this section. 

The PERT system includes two major cycles: the planning 
cycle and the implementation and control cycle. 
The planning cycle includes: 

• definition of project objectives. 

• specification c milestones for the project. 

• breakdown of tasks. 

• further breakdown within organizational departments. 

• definition of resources* ' 

• development of a cost-accounting system. 

• construction of the networks. 

• estimation of time and resources for each task. 

• development of delivery schedules for all milestones. 



The implementation and control cycle includes: 

• development of all data processing requirements. 

• implementation of cost -accounting system. 

• comparison of actual expenditures in time and resources 
to projected expenditures. \ 

• updating of PERT network. 

• revision of schedules and budget. 
THE PLANNING CYCLE 

1. Definition of Project Objcctiv.^ s 

The first step in developing the PERT management 
system is to define the objectives of the project. 
These objectives should include a statement of all 
deliverable documents for the contract. To accomplish 
this step, a careful review was made of the proposal, 
the "Multi -Media Course Development Paper", the "Pro- 
gram Data Book", and all other written and verbal 
communications between WLC and the USNA. 

The multi -media course development project had 
three major objectives: 

a. The development of a multi -media course in leader- 
ship for the USNA. This course was to include the 
most recent advances in the field of educational 
technology and was to be fully validated in the 
classroom. 

b. A comprehensive research program to evaluate the 
effectiveness of various media and presentational 
forms with respect to variables including' cost. 



student achievement, and administrative difficulties. 
Relationships between individual learning styles and 
subsequent performance in the instructional environ- 
ment were to be carefully studied. The outcome of 
the research effort would be the establishment of 
guidelines for use in developing the best possible 
prescription for any student, based on his individual 
characteristics and the possibilities of the course 
of study. 

c. The primary long-range objective was the design, de- 
velopment, and verification of a model which could 
be used for the development of a. superior educational 
system for secondary schools. 
Specification of Milestones 

The major milestones for Phase I were initially 
defined as: (1) the research and evaluation plan, (2) 
the content outline, (3) the terminal objectives, (4) the 
enabling objectives and test items, (5) the presentation 
design, (6) the segment documentation packages, and (7) 
the actual material for the first one-half of the course 
(this included instructional material, student manuals, 
and teachers* guides). 

In addition, progress reports were to be submitted 
monthly and quarterly. Additional milestones were 
believed to be important for the fulfillment of the con- 
tract objectives, but were not part of the critical path 
for the workflow of course construction and the research 
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effort. These milestones include: (1) the development 
of a cost-accounting system, (2) the development of a 
rationale for media selection, (3) the development of 
the rationale for sequencing, and (4) a series of tech- 
nical papers which included topics such as course descrip- 
tion, course strategy, course development model, and 
cost effectiveness. 
Breakdown of Tasks 

The specification of tasks served as the basis for 
construction of the PERT network. The actual task break- 
down began with the construction of the course develop- 
ment model. The development of this model and the task 
definition began at the most general functional level. 
The subdivision of the functions continued until a level 
was reached at which the complete resource requirement 
could be completely specified. All tasks necessary for 
the accomplishment of each milestone were delineated. 
Care was taken to ensure that these tasks were con- 
sistent with the work statements presented in the 
original proposal. 

After the tasks were specified, their interrelation- 
ships were defined. 

Up to this point, the ^relationships between the course 
development model and the structure of the PERT network 
were quite similar. At this point, however, it was nec- 
essary to assign responsibility for. the specific tasks to 
departments or units of operation in the customers' 



organization. From this point on, the development of the 
PERT network began to reflect the organizational struc- 
ture and specific resource availability of the customer 
and, therefore, deviated sharply from the general model. 
Task Br eakdown Within Orgni zational Departments 

Each department in the organization identified a 
number of steps within any task that was assigned to it. 
An example of this type of detail is presented in the 
chart on page 28. This chart shows a breakdown of the 
steps involved in the production of a unit of material. 
The extent and number of these smaller subdivisions 
depended upon the amount of detail needed by the depart- 
ment manager to plan and control his work. In this parti- 
cular project, the extremely tight schedule made it nec- 
essary to detail work to the extent th?^ -esources could 
be estimated to the nearest man-day. 

Another variable which influenced the extent to which 
the isteps in any task were identified was the number of 
departments and different personnel in the contractor's 
organization which were responsible for the accomplishment 
of that task. For example, the drafting of a report was 
usually broken down into two smaller steps -- editing and 
typing -- for assignment of responsibility to different 
sections. On the other hand, the development of test 
norms, which was a much larger task, was not broken down 
since the responsibility for the various steps involved 
rested with a single section in the contractor's organiza- 



tion. 



As the structure of the step breakdown is defined, 
the interfaces between any of the various steps should^ 
be identified. Interfaces are defined as the transfer 
of information from one part of the network to another. 
If this transfer of information is shown on the chart 
by a dotted line, the transfer does not consume time or 
resources . 
S. Definition of Resources 

Closely related to the breakdown, with respect to 
organizational structure, is the definition of resources 
The computer program which was developed for the PERT 
Network* allowed for the specification of ten ^pro- 
fessional "resource" categories. 

- These categories included: editor, secretary, pro- 
ject coordinator, writer, analyst, illustrator, research 
assistant, research associate, production and control 
manager, and subject matter expert. For each step in 
the network, the number of man hours required for each 
resource was specified. 

After the specification of resources and man hours 
had been completed, manpower loading for any given pro- 
ject-day was determined. Charts showing the overall 
requirements for each of the professional resources were 
developed. 



let, SexfcT O^^-loP-" the Dikewood Corporation f Albuquerque . 
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Figure 1 on the following page shows the manpower 
for a typing support staff for each project week. 

These charts were analyzed by the management of 
each department to shift resources in order to alleviate 
the problems of excessive manloading or idle manpower. 
Highly irregular loading patterns would indicate a need 
for additional hiring, overtime, or possibly a resched- 
uling of the activities listed in the PERT network. 
Developm ent of a Cost-Accounting System 

The cost-accounting system, developed for the 
multi-media project, has been discussed in detail in 
the technical paper TP-6.5. One aspect of this 
system which was alluded to, but not presented in its 
entirety, was the relationship of the cost-accounting 
system to the PERT network. The code structure for the 
accounting system was devised in such a way that costs 
could be isolated for any professional resource (limited 
to the ten previously mentioned) working on any step in 
the network. This aspect of the project was necessary 
for the feedback, updating, and control of the system. 

All costs could be summarized with respect to any 
given milestone, resource, or organizational department. 
In this way, the actual man hours spent for any activity 
could be directly and immediately compared to the pro- 
jected hours for that activity. . Variations in the actual 
and the projected man hours served as a basis for revi- 
sions of projected costs and milestone schedules. 
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7. Construction of the Network 

The network is the graphic representation of the 
project plan. It shows the plan, which has been estab- 
lished to reach each milestone, along with the inter- 
relationship and interdependence of the project steps. 

A network is composed of events and activities . 
Events is the term used when referring to the start or 
completion of an activity. These do not consume time 
or resources. Events are points in time which signal 
the , initiation or completion of an action. 

Generally, events in a network are represented by 
the use of circles. For purposes of reference, these 
circles were assigned specific numbers in this project. 
In addition, the event was considered to represent the 
completion of the step which directly preceded it. 



For example, in Figure 2, ^lumber 62 represented the 
completion of the development of norms for the tests. 
Number 61 represented the completion of the tests, 
rather than the start of the development of norms. The 
resources, time, and costs of the project were itemized 
by these numbers. An example of a computer print-out 
form is presented in Figure 3> on the following page. 



Develop Tests, (61 




Develop Norms for Test 



Figure 2 - Example of a Portion of the 
PERT Network 
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I An activity is a task or step in the project for 
which the use of specific resources for a period of time 
can be delineated. An activity is the actual work that 
takes place between two events. An activity is repre- 
sented on the network by an arrow connecting one event 
with another. There are also dummy activities which 
are represented by dotted arrows or lines. These ac- 
tivities do not consume time or resources. An example 
of a network showing events, activities and dummy 
activities is shown in Figure 4. Note that the flow 
of the network always moves from right to left. One 
essential requirement of the computer program was that 
the event (or circle) at the beginning of an activity 
must always have a smaller number than the event at 
the end of the activity. The number assigned to the 
events do not necessarily indicate the order in which 
they occurred in the project. For example, in Figure 4, 
event 7 does not necessarily occur after event 8. 



0—0— 0—^0 — 
0"""' 

Figure 4 - Illustration of the Relationship 
of Events to One Another 
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In the P2RT network designed for the multi-media project, 
the use of a number of different types of construction 
was required. 



activities or steps were dependent or progressive • See 
Figure 5 below. 



When two or more activities could have taken place 
at the same time, but were constrained by a common 
previous activity and concluded in a common end activ- 
ity, a parallel construction was used, as in Figure 6 
below. 



A series construction was used when the successive 



0 



0 



-0 



Figure 5 -- Series Construction 




o 



Figure 6 



Parallel Construction 
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Other types of activities are illustrated in Figure 
7 below. While Figure 6 shows several activities which 
are constrained by a preceding event. Figure 7 shows a 
number of activities which end in a common event • 



I 



I 



I 





Figure 7 Activities Beginning or Ending in a Common Event 

A number of variations and combinations of these 
constructions were used. It should be noted that a 
multiple start point could have been used with this net- 
work* It is always necessary to tie these multiple 
beginnings together with dummy arrows so that all 
events be connected to comply with the requirements for 
the data processing system. An additional idiosyncrasy 
of the PERT network, as a product of the data processing 
system, was the use of 999 in the final event circle. 
The particular program developed had the capacity to 
handle 999 events. Consequently, 999 was used to indi- 
cate that a particular activity ended at the' completion 
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of the project (in this case, Phase !)• In some cases 
this eliminated the need for long dummy arrows which 
would detract from the major path of the work flow. 
Estimation of Time and Resource Requirements 

As previously mentioned, the resources or man hours 
were specified for each step or activity in the project 
effort. This, of course, directly effected the times 
which were assigned, but did not completely determine 
them. 

There are two generally accepted methods for estab- 
lishing time estimates for each activity in the network. 
The most common approach requires the establishment of 
three time estimates; the other requires the use of 
only one time estimate. By using the first method, the 
optimistic, most likely, and pessimistic times for each 
activity are specified. The "expected"' time is then 

computed from the formula '"t^ = opt. ♦ 4 (m.l.) ♦ p ", 

6 

For this particular project only one time estimate was 
used. 

Once the time estimates for each step were deter* 
mined, the longest time path through the network was 
calculated. This path is called the critical path, 
since the total project time is dependent on this path. 
All other paths on the network are called slack paths. 
In the PERT network shown in this paper, many of the 
slack paths have been eliminated. These paths consist 
of repetitions of a section of the chart which begins 
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with the preparation of the enabling objectives, and 
ends at the final delivery of materials for various 
chapters. The critical path is shown for chapters X 
and Y. X and Y represent those chapters which require 
the longest time for completion. 

Although only one estimate of time was used, the 
fixed beginning and termination of the contract allowed 
for the computation of slippage time . The time required 
for the completion of the critical path was calculated 
forward from the initial date and backwards from the 
termination date of the contract.. This calculation 
yielded the maximum amount of "slippage'' time that could 
occur without affecting the final delivery date, as 
specified by the contract. 
Development of Delivery Schedul'tes 

By analyzing the manpower loadings and the total 
worktime required for each activity, the delivery sched- 
ule from each milestone was developed. Initially, three 
times were indicated for delivery of each milestone. 

They were: 

• delivery to USNA. 

• return to WLC with recommended revisions. 

• final delivery to USNA. 

All three times were indicated on the schedule 
because the amount of turn-around time alloted for the 
USNA to review materials was critical to the completion 
of the contract. Figure 8, on page 22, shows a 
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hypothetical schedule which could have been developed 

for the multi -media materials development contract* 
THE IMPLEMENTATION AND CONTROL CYCLE 

The control cycle for any project is one of the most 
important functions of the PERT network • for the multi- 
media course development project, a special cost-accounting 
system was developed by WLC in such a way that it could 
satisfy corporate budget processing requirements and feed 
directly into the PERT system* Each activity in the PERT 
network was assigned a unique budget number* This number 
identified the activity and the functional group to w^ich 
it belonged* In addition, the charge numbers used od time 
cards reflected the resource which was involved in the 
activity* Direct labor costs, travel, living expenses, 
etc*, were all charged to a specific activity* With this 
system, a weekly read-out was obtained which listed the 
expenditures to date for each activity* These expenditures 
could be compared to the predicted man-hour requirements for 
the project* In this way, the network was continually re- 
viewed and updated « 

New account numbers and new activities could be assigned 
whenever necessary* The entire schedule could be changed in 
a few minutes. The computer program which was used to manip 
ulate the PERT network was not sophisticated enough to up- 
date the network when major task changes were made* This 
was especially true of those changes affecting the critical 
path of the network* 
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Schedule for Document Delivery 
Leadership - Phase 1 



I Document 
TR-6.1 
lTR-6.2 



ItR-6.3 

I 

ITR-6.4 



I TR-6.12 



8/30 



9/6 



9/13 



9/20 



w w w 



9/27 



10/4 



10/1] 



10/18 



'\l7 Delivery to USNA 
I \27 Return to WLC with recommended revisions 
I \^ Final delivery to USNA 



10/25 



11/11 



11/18 11/15 



11/2211/; 



27 



Figure 8 -- Example Schedule of Milestones 
for Phase I of the USNA Multi- 
Media Project 
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In general, the control cycle can be summarized in four 
major steps. 

• Implementations of the cost-accounting system 

• Comparisons of actual expenditures in time and 
.resources to projected expenditures 

• Continual updating of the PERT network 

• Revision of the milestone schedules and budget for 
the project 

Although the initial development and implementation of 
the system required an extensive effort, the automation of 
the control cycle alloved for a great deal of information to 
be processed with a limited amount of time and expense. 
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III. EVALUATION OF THE SYSTEM 

The PERT management network represented an ideal system for 
the planning and control of the project. However, five months 
after the beginning of Phase I, it became increasingly evident to 
WLC that the use of the PERT system for allocation of resources 
and estimation of deadlines was no longer effective. After a 
careful examination of the project, there appeared to be four 
factors which continually caused problems in the management of 
the effort and eventually led to the abandonment of the PERT/ 
cost system. It should be emphasized at this time that it is 
not the purpose of this paper to be critical of any element 
involved in the multi-media course development project. Rather, 
it is hoped that ir* pointing out the problem areas, potential 
difficulties with future projects of this type can be avoided. 
A. CONTROL OF ACTIVITIES 

A management system such as PERT assumes that all acti- 
vities can be assigned reasonable estimates of time and 
resources. It also assumes that all activities can be 
brought under the control of the contractor by manipulating 
the resources or time involved. In most developmental pro- 
jects where PERT has beer^ used successfully, the interaction 
with the contractor, i.e., review points, occurs at the termin- 
ation of a long chain of activities. In a materials develop- 
ment project such as the multi-media project, continued 
interaction with the contractor was required. 

One way to look at this problem is to consider the 
difficulty that would occur if one department in an organiza- 
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tion was not scheduled by the PERT system but had a vital 
part in the development of the materials. The requirement 
for continual interaction with the customer and multiple 
sign-off points made the accurate specifications of the 
customers time on the project essential. Without this con- 
trol, the careful scheduling of resources and time around 
undefinable blocks of time was not feasible. 

In order to efficiently schedule a project of this type, 
a contractual aggreement would need to be established which 
would specify the amount of turn-around time necessary for 
each submittal. The establishment of such events, prior to 
the project start, would be advantageous to both the customer 
and the contractor. 
B. DEPENDENCY OF APPROVALS 

A second problem which served as both the major reason 
for the development of the PERT system and the major reason 
for its abandonment was the interdependency of the mile- 
stones. In almost every case, the approval of one milestone 
was necessary before the work on the following milestone was 
begun. This requirement, in combination with a limited time 
deadline, posed a major problem for both the contractor and 
the customer. For example, the course outline served as a 
basis for the terminal objectives, which in turn served as a 
basis for the evaluating objectives, etc., on through the pre- 
sentation design, segment documentation, and materials pre- 
paration. Therefore, the approval of the course outline was 
essential for work to progress from the contractor's point of 
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view. The customer, however, was hesitant in making a 
hasty decision on a document which would serve as a basis 
for the entire project. This problem was never resolved. 
Consequently, milestones were not approved and dependent 
milestones were delayed. The first milestone of the project, 
the content outline, was informally submitted in August, 
1968, and after multiple revisions, it has not yet been 
completely approved, as of the writing of this report. 
May, 1969. 

IVhen this type of dependency relationship is inherent 
in a pi-oject, an agreement should be reached between the 
customer and the contractor as to the maximum time allotted 
for approval and revision before the final delivery dates 
and project costs are significantly affected. The slack 
time in the PERT network is an excellent way to estimate 
this type of effect. 
NATURE OF THE COURSE 

The third problem is more specific to this project and 
may have limited potential for generalization to other 
projects. It will be mentioned, therefore, but not con- 
sidered in any detail. The problem concerns the content of 
the course itself: "What is leadership?" A simple answer to 
this question is impossible. Leadership is a complex con- 
cept which can be defined in many different ways. This lack 
of tangibility caused many of the time delays which have 
been discussed in this section. 
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NATURE OF THE CONTRACT 

The management system developed for a specific project 
is a function of the precision and clarity of the original 
specification of the work to be completed. Because of the 
lack of clarity of the initial contract, the PERT network was 
based on milestones which the contractor believed to be the 
best representation of the customer's requirements; however, 
agreement was never officially reached as to the items which 
wo/ld constitute total delivery for Phase I of the project. 
This lack of agreement on project milestones, which formed 
the basis for the PERT network, caused major changes in the 
actual path of the system. Since this type of change could 
not be handled by the data processing program, the man hours 
required for rescheduling all activities became -prohibitive. 
The major value of the system was lost when deviations from 
the initial path were so extensive that they could not be 
automatically updated. 

The solution to this problem is simple in theory, but 
difficult in practice* A clear statement of the work to 
be accomplished and the milestones to be reviewed by the 
customer should be included in the contractual agreement. 
This is especially true if the project is a developmental 
one and if the contract references multiple source documents 
which yield varying information concerning the project. 
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IV. CHART SHOWING BREAKDOWN OF 
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